Information processing apparatus, control method, and storage medium

ABSTRACT

An information processing apparatus includes a communication unit configured to communicate with a camera configured to capture a third image which is one image in which left and right images are arranged, and a control unit configured to display the third image on a display unit and set a position of an enlarged range to which enlargement processing is to be applied by the camera in the third image displayed on the display unit, and the control unit converts the set position of the enlarged range based on a display format in which the third image is displayed on the display unit, and outputs an instruction to the camera.

BACKGROUND Technical Field

The aspect of the embodiments relates to an information processingapparatus, a control method, and a storage medium.

Description of the Related Art

There has been known a technique of capturing two images with parallaxusing two cameras, and displaying the captured two images in astereoscopically-viewable manner Japanese Patent Application Laid-OpenNo. 2013-141052 discusses a camera that is attached with a lens unitincluding two optical systems and can capture two images with parallaxat one time.

In some cases, a live view image captured by a digital camera istransmitted to an external terminal of the digital camera, and the liveview image is displayed on a display unit of the external terminal. Theexternal terminal can also control operations of the digital camera bytransmitting a recording start instruction or a control command forimage processing to the digital camera. As the external terminal, apersonal computer, a smartphone, or a tablet that can display an imageon a display is used. A user can check a live view image and control thedigital camera using the external terminal from a position distant fromthe digital camera.

If an image (one image including two images with parallax) captured by adigital camera attached with a lens unit including two optical systemsis displayed in the same way as a conventional image, a positionalrelationship between the two optical systems and a positionalrelationship between the two images in the one image reverse in somecases. Thus, in the case of controlling the digital camera whiledisplaying a live view image acquired from the digital camera on theexternal terminal, image processing different from that executed on animage captured by a digital camera attached with a conventional singlelens is required. Nevertheless, in the prior art such as the techniquediscussed in Japanese Patent Application Laid-Open No. 2013-141052, suchan issue has not been considered enough.

SUMMARY

According to an aspect of the embodiments, a processing apparatusincludes a communication unit configured to communicate with an imagingapparatus configured to capture one third image including a first imagecorresponding to a first image input via a first optical system, and asecond image corresponding to a second image input via a second opticalsystem having predetermined parallax with respect to the first opticalsystem, a control unit configured to display the third image on adisplay unit, and a setting unit configured to set a position of atarget region to which predetermined image processing is to be appliedby the imaging apparatus, in the displayed third image, wherein thesetting unit converts the position of the target region set in the thirdimage displayed on the display unit based on a display format in whichthe control unit displays the third image on the display unit, andwherein the communication unit outputs the converted position of thetarget region to the imaging apparatus.

Further features of the disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are schematic diagrams illustrating an overallconfiguration of a system.

FIGS. 2A and 2B are external views of a camera.

FIG. 3 is a block diagram illustrating a configuration of the camera.

FIG. 4 is a schematic diagram illustrating a configuration of a lensunit.

FIG. 5 is a block diagram illustrating a configuration of a personalcomputer (PC).

FIG. 6 is a flowchart illustrating an operation of the camera.

FIGS. 7A and 7B are schematic diagrams illustrating an example of lensinformation to be acquired from dual lenses.

FIG. 8 is a flowchart illustrating an example of an operation of the PC.

FIGS. 9A and 9B are schematic diagrams of left-right swapping.

FIG. 10 is a schematic diagram illustrating equirectangular conversionincluding left-right swapping conversion.

FIGS. 11A, 11B, and 11C are schematic diagrams each illustrating adisplay example of PC live view.

FIGS. 12A, 12B, and 12C are schematic diagrams each illustrating adisplay example of PC live view to be displayed in a case where anenlarged image is displayed.

FIG. 13 is a flowchart illustrating display processing of an enlargedimage.

FIGS. 14A, 14B, and 14C are schematic diagrams illustrating anenlargement frame movement command.

FIG. 15 is a flowchart illustrating enlargement frame movementprocessing in PC live view display of displaying a non-enlarged image.

FIG. 16 is a flowchart illustrating enlargement frame movementprocessing in PC live view display of displaying an enlarged image.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, an exemplary embodiment of the disclosure will be describedin detail with reference to the drawings.

An exemplary embodiment of the disclosure will be described. FIGS. 1Aand 1B are schematic diagrams each illustrating an example of an overallconfiguration of a system according to the present exemplary embodiment.The system according to the present exemplary embodiment includes adigital camera (camera) 100 and a personal computer (PC) 500. A lensunit 300 is attached (connected) to the camera 100. The details of thelens unit 300 will be described below. Being attached with the lens unit300, the camera 100 can capture two images (still images or movingimages) with predetermined parallax at one time. The PC 500 is aninformation processing apparatus that handles images captured by animaging apparatus such as the camera 100. FIG. 1A illustrates aconfiguration in which the camera 100 and the PC 500 are connectedwirelessly or by wire so that communication can be performedtherebetween. FIG. 1B illustrates a configuration in which imagescaptured by the camera 100 are input to the PC 500 via an externalstorage device on a file-basis. The external storage device may beconnected to both the camera 100 and the PC 500, or the external storagedevice may be connected to either the camera 100 or the PC 500. Forexample, the external storage device may be connected to the camera 100,and files of images captured by the camera 100 may be stored in theexternal storage device. After that, the external storage device may bedetached from the camera 100 and connected to the PC 500, and the PC 500may import the files stored in the external storage device.

FIGS. 2A and 2B are external views illustrating an example of theexternal appearance of the camera 100. FIG. 2A is a perspective view ofthe camera 100 viewed from the front side, and FIG. 2B is a perspectiveview of the camera 100 viewed from the rear side.

On the top surface, the camera 100 includes a shutter button 101, apower switch 102, a mode selection switch 103, a main electronic dial104, a sub electronic dial 105, a moving image button 106, and anextra-viewfinder display unit 107. The shutter button 101 is anoperation member for issuing an imaging preparation instruction or animaging instruction. The power switch 102 is an operation member forswitching between power-on and power-off of the camera 100. The modeselection switch 103 is an operation member for switching betweenvarious modes. The main electronic dial 104 is a rotary operation memberfor changing a setting value such as a shutter speed and an aperturevalue. The sub electronic dial 105 is a rotary operation member formoving a selection frame (cursor) and performing image feeding. Themoving image button 106 is an operation member for issuing a start orstop instruction of moving image capturing (recording). Theextra-viewfinder display unit 107 displays various setting values suchas a shutter speed and an aperture value.

On the rear surface, the camera 100 includes a display unit 108, a touchpanel 109, a directional key 110, a SET button 111, an autoexposure (AE)lock button 112, an enlargement button 113, a reproduction button 114, amenu button 115, an eyepiece unit 116, an eye approach detection unit118, and a touch bar 119. The display unit 108 displays an image andvarious types of information. The touch panel 109 is an operation memberfor detecting a touch operation on a display surface (touch operationsurface) of the display unit 108. The directional key 110 is anoperation member including a key that can be pressed upward, downward,leftward, and rightward (four-way key). Processing corresponding to apressed position of the directional key 110 is performed. The SET button111 is an operation member to be pressed mainly to determine a selecteditem. The AE lock button 112 is an operation member to be pressed to fixan exposure state in an imaging standby state. The enlargement button113 is an operation member for switching between on and off of anenlargement mode in live view display (LV display) of an imaging mode.In a case where the enlargement mode is on, a live view image (LV image)is enlarged or reduced by a user operating the main electronic dial 104.The enlargement button 113 is also used to enlarge a reproduced image ina reproduction mode, or to increase an enlargement ratio. Thereproduction button 114 is an operation member for switching between theimaging mode and the reproduction mode. In a case where the camera 100is in the imaging mode, the camera 100 shifts to the reproduction modeif the reproduction button 114 is pressed, and a latest image amongimages recorded on a recording medium 227 to be described below can bedisplayed on the display unit 108.

The menu button 115 is an operation member to be pressed for displaying,on the display unit 108, a menu screen for enabling various settings.The user can intuitively make various settings using the menu screendisplayed on the display unit 108, the directional key 110, and the SETbutton 111. The eyepiece unit 116 is a portion to which an eye of theuser is brought close and through which the user looks into an eyepieceviewfinder (look-in viewfinder) 117. Through the eyepiece unit 116, theuser can view an image displayed on an electronic viewfinder (EVF) 217inside the camera 100, which will be described below. The eye approachdetection unit 118 is a sensor for detecting whether the user's eye hasapproached the eyepiece unit 116 (eyepiece viewfinder 117).

The touch bar 119 is a linear touch operation member (line touch sensor)that can receive a touch operation. The touch bar 119 is arranged at aposition touch-operable (touchable) by a right thumb in a state in whicha grip portion 120 is gripped by a right hand (gripped by a right littlefinger, a right ring finger, and a right middle finger) in such a mannerthat a right index finger can press the shutter button 101. In otherwords, the touch bar 119 is operable in a state in which the user islooking into the eyepiece unit 116 with the eye of the user broughtclose to the eyepiece viewfinder 117, and is holding the camera 100ready to press the shutter button 101 (image capturing orientation). Thetouch bar 119 can receive a tap operation on the touch bar 119 (anoperation of touching the touch bar 119 with a finger and removing thefinger within a predetermined time period without moving a touchposition), and a left-right slide operation on the touch bar 119 (anoperation of touching the touch bar 119 and then moving a touch positionwhile touching the touch bar 119). The touch bar 119 is an operationmember different from the touch panel 109, and does not include adisplay function. The touch bar 119 functions as a multifunction bar(M-Fn bar) to which various functions can be allocated, for example.

The camera 100 further includes the grip portion 120, a thumb restportion 121, a terminal cover 122, a lid 123, and a communicationterminal 124. The grip portion 120 is a holding portion formed into ashape that can be easily gripped by a right hand when the user holds thecamera 100. The shutter button 101 and the main electronic dial 104 arearranged at positions operable by the right index finger in a state inwhich the user holds the camera 100 by gripping the grip portion 120with the right little finger, the right ring finger, and the rightmiddle finger. In addition, the sub electronic dial 105 and the touchbar 119 are arranged at positions operable by the right thumb in asimilar state. The thumb rest portion 121 (thumb standby position) is agrip portion provided at a point where the user can naturally placehis/her thumb of the right hand gripping the grip portion 120 in a statein which the user operates none of the operation members provided on therear side of the camera 100. The thumb rest portion 121 is formed ofrubber member for strengthening holding force (gripping force). Theterminal cover 122 protects a connector of a connection cable thatconnects the camera 100 to an external device (external apparatus). Thelid 123 protects the recording medium 227 and a slot for storing therecording medium 227, which will be described below, by blocking theslot. The communication terminal 124 is a terminal for communicatingwith a lens unit (a lens unit 200 to be described below, the lens unit300, etc.) detachably attached to the camera 100.

FIG. 3 is a block diagram illustrating an example of a configuration ofthe camera 100. The same components as those illustrated in FIGS. 2A and2B are assigned the same reference numerals as those illustrated inFIGS. 2A and 2B, and the description of the components will beappropriately omitted. In FIG. 3 , the lens unit 200 is attached to thecamera 100.

First, the lens unit 200 will be described. The lens unit 200 is onetype of interchangeable lens detachably attached to the camera 100. Thelens unit 200 includes a single lens, and serves as an example of anormal lens. The lens unit 200 includes an aperture 201, a lens 202, anaperture drive circuit 203, an autofocus (AF) drive circuit 204, a lenssystem control circuit 205, and a communication terminal 206.

The aperture 201 has an adjustable aperture diameter. The lens 202includes a plurality of lenses. The aperture drive circuit 203 adjusts alight amount by controlling the aperture diameter of the aperture 201.The AF drive circuit 204 executes focusing by driving the lens 202.Based on an instruction from a system control unit 50 to be describedbelow, the lens system control circuit 205 controls the aperture drivecircuit 203 and the AF drive circuit 204. The lens system controlcircuit 205 controls the aperture 201 via the aperture drive circuit203, and executes focusing by changing the position of the lens 202 viathe AF drive circuit 204. The lens system control circuit 205 cancommunicate with the camera 100. Specifically, communication isperformed via the communication terminal 206 of the lens unit 200 andthe communication terminal 124 of the camera 100. The communicationterminal 206 is a terminal for the lens unit 200 to communicate with thecamera 100.

Next, the camera 100 will be described. The camera 100 includes ashutter 210, an imaging unit 211, an analog-to-digital (A/D) converter212, a memory control unit 213, an image processing unit 214, a memory215, a digital-to-analog (D/A) converter 216, the EVF 217, the displayunit 108, and the system control unit 50.

The shutter 210 is a focal plane shutter that can freely control anexposure time of the imaging unit 211 based on an instruction from thesystem control unit 50. The imaging unit 211 is an image sensorincluding a charge-coupled device (CCD) sensor or a complementarymetal-oxide semiconductor (CMOS) sensor that converts an optical imageinto an electrical signal. The imaging unit 211 may include an imagingplane phase difference sensor for outputting defocus amount informationto the system control unit 50. The A/D converter 212 converts an analogsignal output from the imaging unit 211 into a digital signal. The imageprocessing unit 214 performs predetermined processing (pixelinterpolation, resize processing such as reduction, color conversionprocessing, etc.) on data from the A/D converter 212 or data from thememory control unit 213. The image processing unit 214 also performspredetermined calculation processing using data of a captured image.Based on an obtained calculation result, the system control unit 50performs exposure control and ranging control. By the processing,through-the-lens (TTL) system AF processing, AE processing, andelectronic flash pre-emission (EF) processing are performed. The imageprocessing unit 214 further performs predetermined calculationprocessing using data of a captured image, and the system control unit50 performs TTL system automatic white balance (AWB) processing based onthe obtained calculation result.

Image data from the A/D converter 212 is written into the memory 215 viathe image processing unit 214 and the memory control unit 213.Alternatively, image data from the A/D converter 212 is written into thememory 215 via the memory control unit 213 and not via the imageprocessing unit 214. The memory 215 stores image data obtained by theimaging unit 211 and converted by the A/D converter 212 into digitaldata, and image data to be displayed on the display unit 108 or the EVF217. The memory 215 has a storage capacity sufficient for storing apredetermined number of still images, and a predetermined time length ofa moving image and audio. The memory 215 also serves as a memory (videomemory) for image display.

The D/A converter 216 converts image data for display stored in thememory 215 into an analog signal, and supplies the analog signal to thedisplay unit 108 or the EVF 217. The image data for display that hasbeen written into the memory 215 is accordingly displayed on the displayunit 108 or the EVF 217 via the D/A converter 216. The display unit 108and the EVF 217 perform display corresponding to the analog signal fromthe D/A converter 216. The display unit 108 and the EVF 217 are displayssuch as a liquid crystal display (LCD) and an organicelectroluminescence (EL) display, for example. Digital signals havingbeen once A/D-converted by the A/D converter 212 and stored in thememory 215 are converted into analog signals by the D/A converter 216,and the analog signals are sequentially transferred to the display unit108 or the EVF 217 and displayed thereon. Live view display is therebyperformed.

The system control unit 50 is a control unit including at least oneprocessor and/or at least one circuit. In other words, the systemcontrol unit 50 may be a processor, a circuit, or a combination of aprocessor and a circuit. The system control unit 50 controls the entirecamera 100. By executing a program recorded on a nonvolatile memory 219,the system control unit 50 implements each piece of processing in aflowchart, which will be described below. The system control unit 50also performs display control by controlling the memory 215, the D/Aconverter 216, the display unit 108, and the EVF 217.

The camera 100 further includes a system memory 218, the nonvolatilememory 219, a system timer 220, a communication unit 221, an orientationdetection unit 222, and the eye approach detection unit 118.

For example, a random access memory (RAM) is used as the system memory218. Constants for operating the system control unit 50, variables, andprograms read from the nonvolatile memory 219 are loaded into the systemmemory 218. The nonvolatile memory 219 is an electrically erasable andrecordable memory. For example, an electrically erasable programmableread-only memory (EEPROM) is used as the nonvolatile memory 219.Constants for operating the system control unit 50 and programs arerecorded in the nonvolatile memory 219. The programs refer to programsfor executing flowcharts to be described below. The system timer 220 isa time measuring unit for measuring a time used for various types ofcontrol or a time of a built-in clock.

The communication unit 221 transmits and receives video signals andaudio signals to and from an external device connected wirelessly or bywire. The communication unit 221 can also connect to a wireless localarea network (LAN) or the Internet. The communication unit 221 can alsocommunicate with the external device via Bluetooth® or Bluetooth LowEnergy. The communication unit 221 can transmit an image (including alive image) captured by the imaging unit 211, and an image recorded onthe recording medium 227. The communication unit 221 can also receiveimages and other various types of information from the external device.The orientation detection unit 222 detects the orientation of the camera100 with respect to a direction of gravitational force. Based on theorientation detected by the orientation detection unit 222, it can bedetermined whether an image captured by the imaging unit 211 is an imagecaptured with the camera 100 being held in a traverse direction or animage captured with the camera 100 being held in a longitudinaldirection. The system control unit 50 can add orientation informationcorresponding to the orientation detected by the orientation detectionunit 222 to an image file of an image captured by the imaging unit 211,or rotate an image to suit the detected orientation. As the orientationdetection unit 222, for example, an acceleration sensor or a gyroscopesensor can be used. Using the orientation detection unit 222, themovement of the camera 100 (whether the camera 100 is panning, tilting,lifted, or still, etc.) can also be detected.

The eye approach detection unit 118 can detect the approach of an objectof some kind to the eyepiece unit 116 (the eyepiece viewfinder 117). Forexample, an infrared light proximity sensor can be used as the eyeapproach detection unit 118. In a case where an object approaches theinfrared light proximity sensor, infrared light projected from a lightprojection unit of the eye approach detection unit 118 is reflected bythe object, and the reflected light is received by a light receivingunit of the infrared light proximity sensor. Based on an amount ofreceived infrared light, a distance to the object from the eyepiece unit116 can be determined. In this manner, the eye approach detection unit118 performs eye approach detection of detecting a near distance of anobject to the eyepiece unit 116. The eye approach detection unit 118 isan eye approach detection sensor for detecting approach (eye approach)and separation (eye withdrawal) of an eye (object) with respect to theeyepiece unit 116. In a case where an object that approaches theeyepiece unit 116 and falls within a predetermined distance from theeyepiece unit 116 is detected from a non-eye approach state(non-approach state), the eye approach detection unit 118 detects thatan eye is in proximity to the eyepiece unit 116. On the other hand, in acase where an object detected to be in proximity to the eyepiece unit116 is separated from the eyepiece unit 116 by a predetermined distanceor more from the eye approach state (approach state), the eye approachdetection unit 118 detects that the eye has been withdrawn.

A threshold for detecting eye approach and a threshold for detecting eyewithdrawal may be different from each other by providing a hysteresis,for example. After the eye approach is detected, the eye stays in theeye approach state until eye withdrawal is detected. After the eyewithdrawal is detected, the eye stays in the non-approach state untileye approach is detected. Depending on the state detected by the eyeapproach detection unit 118, the system control unit 50 switches thedisplay (displayed state) and nondisplay (non-displayed state) of thedisplay unit 108 and the EVF 217. Specifically, at least in a case wherethe camera 100 is in an imaging standby state and a switching setting ofa display destination is set to an automatic switching setting, in thenon-eye approach state, the display destination is set to the displayunit 108 and the display is set to on, and the EVF 217 is brought intothe non-displayed state. In contrast, in the eye approach state, thedisplay destination is set to the EVF 217 and the display is set to on,and the display unit 108 is brought into the non-displayed state. Theeye approach detection unit 118 is not limited to an infrared lightproximity sensor, and another sensor may be used as the eye approachdetection unit 118 as long as the sensor can detect a state that can beregarded as eye approach.

The camera 100 further includes the extra-viewfinder display unit 107,an extra-viewfinder display drive circuit 223, a power source controlunit 224, a power source unit 225, a recording medium interface (I/F)226, and an operation unit 228.

The extra-viewfinder display unit 107 is driven by the extra-viewfinderdisplay drive circuit 223, and displays various setting values of thecamera 100 such as a shutter speed and an aperture value. The powersource control unit 224 includes a battery detection circuit, a directcurrent (DC)-DC converter, and a switch circuit for switching a block tobe supplied with power. The power source control unit 224 detectswhether a battery is attached, the type of battery, and remainingbattery capacity. The power source control unit 224 controls the DC-DCconverter based on the detection result and an instruction from thesystem control unit 50, and supplies voltage to components including therecording medium 227 for a time period. The power source unit 225includes a primary battery such as an alkaline battery or a lithiumbattery, a secondary battery such as a nickel-cadmium (NiCd) battery, anickel-metal hydride (NiMH) battery, or a lithium (Li) battery, and analternating current (AC) adapter. The recording medium I/F 226 is aninterface with the recording medium 227 such as a memory card or a harddisc. The recording medium 227 is a memory card or the like forrecording a captured image, and includes a semiconductor memory or amagnetic disc. The recording medium 227 may be detachably attached tothe camera 100, or may be built into the camera 100.

The operation unit 228 is an input unit for receiving operations fromthe user (user operations), and is used for inputting variousinstructions to the system control unit 50. The operation unit 228includes the shutter button 101, the power switch 102, the modeselection switch 103, the touch panel 109, and other operation units229. The other operation units 229 include the main electronic dial 104,the sub electronic dial 105, the moving image button 106, thedirectional key 110, the SET button 111, the AE lock button 112, theenlargement button 113, the reproduction button 114, the menu button115, and the touch bar 119.

The shutter button 101 includes a first shutter switch 230 and a secondshutter switch 231. The first shutter switch 230 is turned on in themiddle of an operation of the shutter button 101, i.e., the firstshutter switch 230 is turned on by what is called a half press (animaging preparation instruction), and outputs a first shutter switchsignal SW1. In response to the first shutter switch signal SW1, thesystem control unit 50 starts imaging preparation processing such as theAF processing, AE processing, AWB processing, or EF processing. Thesecond shutter switch 231 is turned on upon completion of an operationof the shutter button 101, i.e., the second shutter switch 231 is turnedon by what is called a full press (imaging instruction), and outputs asecond shutter switch signal SW2. In response to the second shutterswitch signal SW2, the system control unit 50 starts a series of imagingprocesses starting from signal readout from the imaging unit 211 up towriting of a generated image file including a captured image to therecording medium 227.

The mode selection switch 103 switches an operation mode of the systemcontrol unit 50 to one of a still image capturing mode, a moving imagecapturing mode, and a reproduction mode. The still image capturing modeincludes modes such as an automatic imaging mode, an automatic scenedetermination mode, a manual mode, an aperture priority mode (Av mode),a shutter speed priority mode (Tv mode), and a program AE mode (P mode).The still image capturing mode further includes modes such as variousscene modes having different imaging settings for respective imagingscenes, and a custom mode. Via the mode selection switch 103, the usercan directly switch the operation mode to any of the above-describedimaging modes. Alternatively, the user can switch the operation mode inthe following manner using the mode selection switch 103, the user onceswitches a screen to a list screen of the imaging modes. Then, using theoperation unit 228, the user can selectively switch the operation modeto any of a plurality of displayed modes. In a similar manner, themoving image capturing mode may include a plurality of modes.

The touch panel 109 is a touch sensor that detects various touchoperations on a display surface of the display unit 108 (operationsurface of the touch panel 109). The touch panel 109 and the displayunit 108 can be integrally formed. For example, the touch panel 109 isattached to the top layer of the display surface of the display unit 108in such a manner that light transmittance does not disturb displayperformed on the display unit 108. Then, an input coordinate on thetouch panel 109 and a display coordinate on the display surface of thedisplay unit 108 are associated with each other. This structure canprovide a graphical user interface (GUI) that performs display as if theuser could directly operate a screen displayed on the display unit 108.As the touch panel 109, a touch panel of any type among the followingvarious types can be used: a resistive touch panel, a capacitive touchpanel, a surface acoustic wave touch panel, an infrared touch panel, anelectromagnetic induction type touch panel, an image recognition typetouch panel, and an optical sensor type touch panel.

Depending on the type, a touch panel may detect a touch by detectingcontact with the touch panel 109 while another touch panel may detect atouch by detecting approach of a finger or a stylus to the touch panel109. A touch panel of any type may be used.

The system control unit 50 can detect the following operations performedon the touch panel 109 or states thereof.

An operation of a finger or a stylus that has not been in touch with thetouch panel 109 newly touching the touch panel 109, i.e., the start of atouch on the touch panel 109 (hereinafter, referred to as Touch-Down).

A state in which a finger or a stylus is in touch with the touch panel109 (hereinafter, referred to as Touch-On).

An operation of a finger or a stylus moving over the touch panel 109while being in touch with the touch panel 109 (hereinafter, referred toas Touch-Move).

An operation of removing (releasing) a finger or a stylus that has beenin touch with the touch panel 109 from the touch panel 109, i.e., theend of a touch on the touch panel 109 (hereinafter, referred to asTouch-Up).

A state in which nothing touches the touch panel 109 (hereinafter,referred to as Touch-Off).

If the Touch-Down is detected, the Touch-On is simultaneously detected.After the Touch-Down, normally, the Touch-On continues to be detecteduntil the Touch-Up is detected. The Touch-On is simultaneously detectedin a case where the Touch-Move is detected. Even if the Touch-On isdetected, the Touch-Move is not detected unless a touch position moves.After the Touch-Up of all the fingers or styluses that have been intouch is detected, the Touch-Off is detected.

The operations and states, and a position coordinate on the touch panel109 at which a finger or a stylus is in touch are notified to the systemcontrol unit 50 via an internal bus. Based on the notified information,the system control unit 50 determines the type of an operation (touchoperation) performed on the touch panel 109. As for the Touch-Move, amoving direction of a finger or a stylus moving on the touch panel 109can be determined for each of vertical and horizontal components on thetouch panel 109 based on a change in position coordinate. In a casewhere it is detected that the Touch-Move is performed for apredetermined distance or more, it is determined that a slide operationhas been performed. An operation of swiftly moving a finger by a certainamount of distance with the finger being in touch with the touch panel109, and removing the finger in this state will be referred to as aflick. In other words, the flick is an operation of swiftly moving thefinger over the touch panel 109 like a flip. If it is detected that theTouch-Move has been performed at a predetermined speed or more for apredetermined distance or more, and the Touch-Up is detected in thisstate, it is determined that a flick has been performed (it can bedetermined that a flick has been performed subsequent to the slideoperation). Furthermore, a touch operation of touching a plurality ofpoints (e.g. two points) concurrently (multi-touch), and bringing thetouch positions closer to each other will be referred to as pinch-in,and a touch operation of bringing the touch positions away from eachother will be referred to as pinch-out. The pinch-out and the pinch-inwill be collectively referred to as a pinch operation (or simply aspinch).

FIG. 4 is a schematic diagram illustrating an example of a configurationof the lens unit 300. FIG. 4 illustrates a state in which the lens unit300 is attached to the camera 100. Among the components of the camera100 that are illustrated in FIG. 4 , the same components as thoseillustrated in FIG. 3 are assigned the same reference numerals as thoseillustrated in FIG. 3 , and the description of the components will beappropriately omitted.

The lens unit 300 is one type of interchangeable lens that is detachablyattached to the camera 100.

The lens unit 300 corresponds to dual lenses that can capture a rightimage and a left image with parallax. In the present exemplaryembodiment, the lens unit 300 includes two optical systems, and each ofthe two optical systems can capture an image in a range with a wideviewing angle of about 180 degrees. Specifically, each of the twooptical systems of the lens unit 300 can capture an image of subjectsexisting in a viewing field (field angle) corresponding to 180 degreesin a left-right direction (horizontal angle, azimuth angle, yaw angle)and 180 degrees in an up-down direction (a vertical angle,elevation/depression angle, pitch angle). In other words, each of thetwo optical systems can capture images in a range of a hemisphere towardthe front.

The lens unit 300 includes a right eye optical system 301R including aplurality of lenses and a reflection mirror, a left eye optical system301L including a plurality of lenses and a reflection mirror, and a lenssystem control circuit 303. The right eye optical system 301R is anexample of a first optical system and the left eye optical system 301Lis an example of a second optical system. The right eye optical system301R includes a lens 302R arranged on a subject side, and the left eyeoptical system 301L includes a lens 302L arranged on the subject side.The lens 302R and the lens 302L are oriented in the same direction, andoptical axes thereof are approximately parallel.

The lens unit 300 corresponds to dual lenses (VR180 lens) for obtainingan image of VR180, which is one of formats of virtual reality (VR)images that enable dual-lens stereopsis. In the present exemplaryembodiment, the right eye optical system 301R and the left eye opticalsystem 301L in the lens unit 300 each includes a fisheye lens that cancapture an image in the range of approximately 180 degrees.Alternatively, the range that can be covered by the lens included ineach of the right eye optical system 301R and the left eye opticalsystem 301L may be about 160 degrees, which is narrower than the rangeof 180 degrees. The lens unit 300 can form a right image (first image)formed via the right eye optical system 301R, and a left image (secondimage) formed via the left eye optical system 301L, on one or two imagesensors of a camera to which the lens unit 300 is attached. An image inwhich the first image and the second image obtained via the lens unit300 are arranged side by side will be referred to as a dual-lens image.

The lens unit 300 is attached to the camera 100 via a lens mount portion304 and a camera mount portion 305 of the camera 100. The system controlunit 50 of the camera 100 and the lens system control circuit 303 of thelens unit 300 are thereby electrically connected via the communicationterminal 124 of the camera 100 and a communication terminal 306 of thelens unit 300.

In the present exemplary embodiment, the right image formed via theright eye optical system 301R, and the left image formed via the lefteye optical system 301L are simultaneously formed (as a set) on theimaging unit 211 of the camera 100. In other words, two optical imagesformed by the right eye optical system 301R and the left eye opticalsystem 301L are formed on one image sensor. The imaging unit 211converts a formed subject image (optical signal) into an analogelectrical signal. By using the lens unit 300 in this manner, from twopoints (optical systems) corresponding to the right eye optical system301R and the left eye optical system 301L, two images with parallax canbe simultaneously acquired (as a set). By VR-displaying the acquiredimages separately as an image for a left eye and an image for a righteye, the user can view a stereoscopic VR image in the range ofapproximately 180 degrees. In other words, the user can stereoscopicallyview an image of VR180.

The VR image refers to an image that can be subjected to VR display,which will be described below. VR images include an omnidirectionalimage (360-degree image) captured by an omnidirectional camera(360-degree camera) and a panorama image having an image range(effective image range) wider than a display range that can be displayedon a display unit at one time. The VR images are not limited to a stillimage, and also includes a moving image and a live image (image acquiredin almost real time from a camera). The VR image has an image range(effective image range) corresponding to a viewing field of 360 degreesin the left-right direction and 360 degrees in the up-down direction atthe maximum. The VR images also include an image having a broader fieldangle than a field angle in which a normal camera can perform imagecapturing, and an image having a wider image range than a display rangethat can be displayed on a display unit at one time, even if the viewingfields of the images are less than 360 degrees in the left-rightdirection and less than 360 degrees in the up-down direction. An imagecaptured by the camera 100 using the above-described lens unit 300 isone type of VR image. The VR image can be VR-displayed by setting adisplay mode of a display device (display device that can display VRimages) to “VR view”, for example. By VR-displaying a VR image having afield angle of 360 degrees and the user changing the orientation of thedisplay device in the left-right direction (horizontal rotationdirection), the user can view an omnidirectional image seamless in theleft-right direction.

The VR display (VR view) refers to a display method (display mode) witha changeable display range that displays an image within a viewing fieldrange corresponding to the orientation of the display device, of the VRimage. The VR display includes “monocular VR display (monocular VRview)” of displaying one image by performing deformation (distortioncorrection) of mapping a VR image on a virtual sphere. The VR displayalso includes “dual-lens VR display (dual-lens VR view)” of displaying aVR image for a left eye and a VR image for a right eye side by side inleft and right regions after performing deformation of mapping the VRimages on the respective virtual spheres. By performing “dual-lens VRdisplay” using the VR image for the left eye and the VR image for theright eye that have parallax, the VR images can be stereoscopicallyviewed. In both types of VR display, for example, in a case where theuser wears a display device such as a head-mounted display (HMD), animage in a viewing field range corresponding to the orientation of theface of the user is displayed. For example, of a VR image, an image in aviewing field range centered on 0 degree in the left-right direction (aspecific direction, for example, north) and 90 degrees in the up-downdirection (90 degrees from the zenith, i.e., horizontal) is displayed ata certain time point. If the orientation of the display device isinverted from this state (for example, the orientation of a displaysurface is changed from south to north), of the same VR image, thedisplay range is changed to an image in a viewing field range centeredon 180 degrees in the left-right direction (an opposite direction, forexample, south) and 90 degrees in the up-down direction. Morespecifically, if the face turns southward from the north (i.e., turnsrearward) in a state in which the user wears an HMD, an image displayedon the HMD is also changed from a north-facing image to a south-facingimage. A VR image captured using the lens unit 300 according to thepresent exemplary embodiment is a captured image in the range of about180 degrees toward the front, and an image in the range of about 180degrees toward the rear does not exist. When such an image isVR-displayed, in a case where the orientation of a display device ischanged to a side on which no image exists, a blank region is displayed.

By VR-displaying a VR image in this manner, the user can visually feelas if the user be inside the VR image (VR space) (immersive feeling).The display method of the VR image is not limited to a method ofchanging the orientation of a display device. For example, a displayrange may be moved (scrolled) in response to a user operation performedvia a touch panel or a directional button. During VR display (when thedisplay mode is set to “VR view”), a display range may be changed inresponse to Touch-Move on a touch panel, a drag operation performed witha mouse, or a press of a directional button, in addition to the changeof the display range caused by an orientation change. A smartphoneattached to VR goggles (head mount adapter) is one type of HMD.

FIG. 5 is a block diagram illustrating an example of a configuration ofthe PC 500. A control unit 501 is a central processing unit (CPU), forexample, and controls the entire PC 500. A read only memory (ROM) 502stores programs and parameters in a non-transitory manner A randomaccess memory (RAM) 503 temporarily stores programs and data suppliedfrom an external device. A recording medium 504 is a hard disc or aflash memory fixedly installed in the PC 500, or an optical disc, amagnetic card, an optical card, an integrated circuit (IC) card, or amemory card that is detachably attached to the PC 500. A file of animage captured by the camera 100 is read from the recording medium 504.An operation unit 505 receives a user operation performed on the PC 500.An operation member to be used by the user for performing operations maybe a button or a touch panel that is provided on the PC 500, or may be akeyboard or a mouse detachably attached to the PC 500.

A display unit 506 displays data stored in the PC 500 and data suppliedfrom the outside. The display unit 506 may be part of the PC 500, or maybe a separate display device independent of the PC 500. A communicationunit 507 performs communication with an external device such as thecamera 100. A system bus 508 connects between the components of the PC500 in such a manner that communication can be performed therebetween.

Features of dual-lens images captured by the camera 100 to which thelens unit 300 (dual lenses) is attached will be described. In the caseof the lens unit 200 (normal single lens), an image inverted verticallyand horizontally from an actual view (an image rotated by 180 degrees)is formed on the imaging unit 211. Thus, an image suitable for theactual view is acquired (captured) by rotating the entire formed imageby 180 degrees. On the other hand, in the case of the lens unit 300(dual lenses), a right image and a left image are each formed on theimaging unit 211 while being rotated by 180 degrees from the actualview. Arrangement of the right image and the left image is notspecifically limited. In the present exemplary embodiment, the rightimage is formed on the right side and the left image is formed on theleft side on the imaging unit 211. Then, if the entire formed image(image including the right image and the left image) is rotated by 180degrees as in the case of the lens unit 200 (normal single lens), whilethe right image and the left image each appear in a manner consistentwith an actual view, the positions of the right image and the left imageare swapped. More specifically, a positional relationship between theright and left images is inverted, and an image in which the right imageis arranged on the left side and the left image is arranged on the rightside is captured. For this reason, even if a captured image is displayedas-is (without considering the swapped positions), a stereoscopic viewcannot be obtained. In the present exemplary embodiment, such an imageis enabled to be stereoscopically viewed.

The control according to the present exemplary embodiment will bedescribed. The description will be given of an example in which thecamera 100 and the PC 500 are connected with each other in such a mannerthat communication can be performed therebetween, a live view imagecaptured by the camera 100 is transmitted to the PC 500, and the PC 500displays the live view image on the display unit 506.

FIG. 6 is a flowchart illustrating an example of an operation of thecamera 100. The operation is an operation of transmitting a live viewimage from the camera 100 for displaying the live view image on the PC500. The operation is implemented by the system control unit 50 loadinga program recorded on the nonvolatile memory 219 into the system memory218 and executing the program. For example, if the camera 100 starts up,the operation illustrated in FIG. 6 starts. The operation illustrated inFIG. 6 is an operation for a function of displaying a live view imagecaptured by a camera on a display unit of a PC (PC live view). Theoperation illustrated in FIG. 6 is executed when the camera 100 is in animaging standby state. In a case where a recording start instruction isinput from the PC 500 during operation of the PC live view, still imagecapturing or moving image capturing is executed.

In step S601, the system control unit 50 determines whether the camera100 is compatible with dual lenses (e.g., the lens unit 300). Forexample, the system control unit 50 determines whether a version offirmware of the system control unit 50 is a version compatible with thedual lenses. In a case where it is determined that the camera 100 iscompatible with the dual lenses (YES in step S601), the processingproceeds to step S602. In a case where it is determined that the camera100 is not compatible with the dual lenses (NO in step S601), theprocessing proceeds to step S611.

In step S602, the system control unit 50 determines whether the duallenses are attached to the camera 100. In a case where it is determinedthat the dual lenses are attached (YES in step S602), the processingproceeds to step S603. In a case where it is determined that the duallenses are not attached (NO in step S602), the processing proceeds tostep S611. Also in a case where the dual lenses are attached from astate in which the dual lenses are not attached, the processing proceedsto step S603. In a case where the dual lenses are detached from a statein which the dual lenses are attached, the processing proceeds to stepS611.

In step S603, the system control unit 50 acquires design values of theattached (connected) dual lenses from the dual lenses. The design valuesare design parameters and are to be used in left-right swapping andequirectangular conversion, which will be described below. For example,an image circle position, an image circle diameter, a field angle, and adistortion correction coefficient illustrated in FIG. 7B are acquired.

In step S604, the system control unit 50 acquires individual values ofthe attached (connected) dual lenses from the dual lenses. An individualvalue is a parameter unique to a lens unit, and is a manufacturingerror, for example. For example, an image circle positional shift, anoptical axis tilt, and an image magnification deviation illustrated inFIG. 7B are acquired. In a case where the individual values are used,image processing can be performed more accurately than in a case wherethe design values are used.

Lens information to be acquired from the lens unit 300 will bedescribed.

FIG. 7A is a schematic diagram illustrating an example of lensinformation to be acquired from dual lenses. The lens informationincludes:

1. Lens design value2. Lens individual value3. Lens flag4. Lens focal length, and5. Lens temperature.

The lens design value is a design value for performing aberrationcorrection. In a manufacturing process of dual lenses, an error such asdecentering or tilt of the lenses occurs in each of the two opticalsystems (the left eye optical system 301L and the right eye opticalsystem 301R). If the left-right swapping or equirectangular conversionis performed without considering the error, the quality of dual-lens VRdisplay declines, and good stereoscopic view becomes difficult. The lensindividual value is a measurement result of an error detected in themanufacturing process of the dual lenses. The details of the lens designvalue and the lens individual value will be described below withreference to FIG. 7B.

The lens flag is a flag indicating that dual lenses are attached. Thelens focal length indicates a distance to an image sensor (image formingposition) from a “principal point” being the center of a lens. The lensfocal length may be a parameter common to the two optical systems (theleft eye optical system 301L and the right eye optical system 301R) ofdual lenses, or may be prepared for each optical system. For the systemcontrol unit 50 to perform high-quality dual-lens VR display byaccurately performing the left-right swapping and equirectangularconversion, a minute (highly-precise) lens focal length is required. Thelens temperature indicates the temperature of the dual lenses, and isused for identifying an environmental temperature in image capturing.

FIG. 7B is a schematic diagram illustrating details of the lens designvalue and the lens individual value. In the present exemplaryembodiment, the lens design value and the lens individual value are usedin the left-right swapping and equirectangular conversion.

The lens design value includes:

1. Image circle position2. Image circle diameter3. Field angle, and4. Distortion correction coefficient.

The image circle position indicates an optical axis central coordinateof an optical system in a captured image, and is prepared for each ofthe two optical systems (the left eye optical system 301L and the righteye optical system 301R) of the dual lenses. In other words, the imagecircle position indicates a central coordinate of an image circle(circular fisheye image) formed on an image sensor, and is prepared foreach of the right image and the left image. An origin of a coordinate isset to the center of the image sensor (the center of a captured image),for example. The image circle position includes a coordinate in ahorizontal direction and a coordinate in a vertical direction. Varioustypes of information regarding an optical axis center of an opticalsystem in a captured image can be used as the image circle position. Forexample, a distance to the optical axis center from a predeterminedposition (center or top-left corner) in an image can be used.

The image circle diameter indicates a diameter of the image circle(circular fisheye image) formed on an image sensor.

The field angle indicates a field angle of the image circle (circularfisheye image) formed on an image sensor. The distortion correctioncoefficient indicates a ratio of a design image height to an ideal imageheight of a lens. The distortion correction coefficient may be set foreach image height, and a distortion correction coefficient for an imageheight for which a distortion correction coefficient is unset may becalculated by interpolation calculation that uses a plurality ofdistortion correction coefficients. A polynomial approximating arelationship between the image height and the distortion correctioncoefficient may be set. The image circle diameter, the field angle, andthe distortion correction coefficient may be parameters common to thetwo optical systems (the left eye optical system 301L and the right eyeoptical system 301R) of dual lenses, or may be prepared for each of thetwo optical systems.

When displaying a circular fisheye image, the PC 500 may display a magicwindow on the circular fisheye image. The magic window is a display itemindicating a region to be extracted (first) for monocular VR display.For example, the magic window is displayed based on an image circleposition, an image circle diameter, and a field angle. The displayquality of the magic window can be thereby enhanced. To appropriatelydisplay the magic window, the PC 500 uses an image circle position, animage circle diameter, and a field angle after appropriately editing thevalues. For example, the PC 500 multiplies an image circle position oran image circle diameter by a coefficient.

The lens individual value includes:

5. Image circle positional shift6. Optical axis tilt, and7. Image magnification deviation.These pieces of information are prepared by performing measurement foreach of the two optical systems (the left eye optical system 301L andthe right eye optical system 301R) of the dual lenses.

The image circle positional shift indicates a deviation from a designvalue of the central coordinate of an image circle (circular fisheyeimage) formed on an image sensor. For example, the image circlepositional shift includes a deviation in the horizontal direction and adeviation in the vertical direction. When an origin is set to acoordinate of a design value (two-dimensional coordinate including acoordinate in the horizontal direction and a coordinate in the verticaldirection), a deviation in the horizontal direction is indicated by acoordinate in the horizontal direction, and a deviation in the verticaldirection is indicated by a coordinate in the vertical direction. Theoptical axis tilt indicates a deviation from a design value of thedirection of an optical axis on the subject side. For example, theoptical axis tilt includes a deviation in the horizontal direction and adeviation in the vertical direction. The deviation in each of thedirections is indicated by an angle. The image magnification deviationindicates a deviation from a design value of a size of an image circle(circular fisheye image) formed on an image sensor. The deviation isindicated by a ratio with respect to a design value, for example.

Information included in the lens information is not limited to theabove-described information. For example, the lens information mayinclude boundary positions of a right image and a left image in acaptured image. The boundary position is a position of a rim of acircular fisheye image, for example, and is a position indicated by ashift amount 905, 906, 909, or 910 illustrated in FIG. 9A, which will bedescribed below. The lens information may include a midpoint coordinatebetween a right image and a left image in a captured image. In manycases, the midpoint coordinate coincides with the central coordinate ofa captured image. The lens information may include informationindicating a region of the magic window (for example, a coordinate of atop-left corner of the magic window, a width of the magic window, and aheight of the magic window). The lens information may include correctiondata for enhancing the accuracy of the left-right swapping orequirectangular conversion (for example, correction value obtained bycalibration of dual lenses).

In step S605, the system control unit 50 detects connection of thecamera 100 to the PC 500. In step S606, the system control unit 50receives a PC live view start request from the PC 500. In step S607, thesystem control unit 50 receives a live view image request from the PC500. As described below, the live view image request includesinformation designating resolution (resolution information) of a liveview image to be transmitted. The system control unit 50 executesprocessing in step S609 to transmit a live view image with thedesignated resolution to the PC 500.

In step S608, the system control unit 50 converts the information (lensinformation regarding dual lenses) acquired in steps S603 and S604 sothat the acquired information is suitable for a coordinate system of alive view image. Because an image to be captured (image to be recordedin an image file) and a live view image differ in resolution, theinformation acquired in steps S603 and S604 cannot be directly used inimage processing of a live view image. Thus, in the present exemplaryembodiment, the system control unit 50 converts the lens informationinto information suitable for a coordinate system of a live view image.

In step S609, the system control unit 50 transmits the lens informationconverted in step S608 and a live view image to the PC 500. The systemcontrol unit 50 converts the resolution of the live view image based onthe resolution information acquired in step S607, and transmits the liveview image to the PC 500. In the present exemplary embodiment, thesystem control unit 50 of the camera 100 converts the lens information,but the control unit 501 of the PC 500 may convert the lens information.In this case, unconverted lens information and parameters for conversionof the lens information are transmitted to the PC 500. In step S610, thesystem control unit 50 determines whether to end PC live view. Forexample, in a case where connection between the camera 100 and the PC500 is canceled, or the user issues an end instruction of the PC liveview to the camera 100 or the PC 500, the system control unit 50determines to end the PC live view. In a case where it is determinedthat the PC live view is to be ended (YES in step S610), the operationillustrated in FIG. 6 is ended. In a case where it is determined thatthe PC live view is not to be ended (NO in step S610), the processingreturns to step S607.

In a case where a single lens is attached to the camera 100 (NO in stepS601 or S602), the processing in step S611 is performed. In step S611,the system control unit 50 transmits a live view image captured by thesingle lens to the PC 500. Because the processing in step S611 issimilar to conventional processing of transmitting a live view imagecaptured by a single lens to an external device, a detailed descriptionwill be omitted.

In the present exemplary embodiment, when transmitting a live view imagecaptured by a single lens to the PC 500, the system control unit 50 doesnot acquire information (design value, individual value, etc.) regardingthe attached single lens from the single lens, and does not transmit theinformation to the PC 500, either.

FIG. 8 is a flowchart illustrating an example of an operation of the PC500. The operation is control for executing PC live view display ofdisplaying a live view image of the camera 100 on the PC 500. Theoperation is implemented by the control unit 501 loading a program(application program) recorded on the ROM 502 into the RAM 503 andexecuting the program. For example, if the user issues a startupinstruction of a specific application to the PC 500, the operationillustrated in FIG. 8 starts. The operation illustrated in FIG. 8 is anoperation for a function of displaying a live view image captured by acamera on a display unit of a PC (PC live view).

In step S801, a camera (e.g., the camera 100) is connected to the PC500, and the control unit 501 detects that the camera has been connectedto the PC 500.

In step S802, the control unit 501 determines whether the cameraconnected in step S801 is a camera compatible with dual lenses (e.g.,the lens unit 300). For example, the control unit 501 acquires modelinformation of the connected camera from the camera, and determineswhether the camera is a camera compatible with dual lenses based on theacquired model information. In a case where it is determined that thecamera is compatible with dual lenses (YES in step S802), the processingproceeds to step S803. In a case where it is determined that the camerais incompatible with dual lenses (NO in step S802), the processingproceeds to step S821.

The camera compatible with dual lenses is a camera to which the duallenses can be attached, for example.

In step S803, the control unit 501 determines whether firmware of thecamera connected in step S801 is compatible with dual lenses. Forexample, the control unit 501 acquires information regarding a versionof firmware of the connected camera from the camera, and determineswhether the version of the firmware of the connected camera is a versioncompatible with dual lenses, based on the acquired information. In acase where it is determined that the firmware is compatible with duallenses (YES in step S803), the processing proceeds to step S804. In acase where it is determined that the firmware is incompatible with duallenses (NO in step S803), the processing proceeds to step S821.

Even if a camera compatible with dual lenses is connected to the PC 500,the connected camera sometimes cannot handle dual lenses due to such areason that the version of the firmware of the connected camera is old.For this reason, the processing in step S803 is to be performed. Inaddition, various cameras can be connected to the PC 500, and a cameraincompatible with dual lenses irrespective of the version of thefirmware is sometimes connected. Thus, the processing in step S802 is tobe performed before the processing in step S803.

In step S804, the control unit 501 determines whether dual lenses areattached to the camera connected in step S801. In a case where it isdetermined that the dual lenses are attached (YES in step S804), theprocessing proceeds to step S805. In a case where it is determined thatthe dual lenses are not attached (NO in step S804), the processingproceeds to step S821.

In step S805, the control unit 501 transmits the PC live view startrequest to the camera connected in step S801.

In step S806, the control unit 501 determines whether to performcircular fisheye display. In a case where it is determined that thecircular fisheye display is to be performed (YES in step S806), theprocessing proceeds to step S807. In a case where it is determined thatthe circular fisheye display is not to be performed (in a case whereequirectangular display is to be performed) (NO in step S806), theprocessing proceeds to step S814. In step S806, the control unit 501determines whether to perform the circular fisheye display depending onwhether a radio button 1105 illustrated in each of FIGS. 11A to 11C isin a selected state or an unselected state, for example. The radiobutton 1105 is in the selected state in FIGS. 11A and 11B, and the radiobutton 1105 is in the unselected state in FIG. 11C. In a case where theradio button 1105 is in the selected state, the control unit 501determines to perform the circular fisheye display, and the processingproceeds to step S807. In a case where the radio button 1105 is in theunselected state, the processing proceeds to step S814.

In step S807, the control unit 501 transmits a live view image requestto the camera connected in step S801. In the present exemplaryembodiment, the live view image request transmitted in step S807 is arequest for a live view image with normal resolution. The normalresolution may be 4K resolution, for example.

In step S808, the control unit 501 receives, from the camera connectedin step S801, a live view image captured by the camera and lensinformation regarding the dual lenses attached to the camera. Theresolution of the live view image received in step S808 is the normalresolution. The lens information received in step S808 is informationconverted to be suitable for the received live view image (for example,the lens information converted in step S608 of FIG. 6 ).

In step S830, the control unit 501 determines whether the PC live viewimage acquired from the camera 100 is a whole image or an enlargedimage. In a case where the acquired image is an enlarged image (YES instep S830), the processing proceeds to step S831. In step S831, thecontrol unit 501 executes display processing of an enlarged image. Thedetails of the processing in step S831 will be described below.

In a case where the PC live view image acquired from the camera 100 is awhole image (NO in step S830), the processing proceeds to step S809.

In step S809, the control unit 501 determines whether to execute theleft-right swapping. In a case where it is determined that theleft-right swapping is to be executed (YES in step S809), the processingproceeds to step S810. In a case where it is determined that theleft-right swapping is not to be executed (NO in step S809), theprocessing proceeds to step S812. In step S809, the control unit 501determines whether to execute the left-right swapping based on whether acheckbox 1107 illustrated in FIGS. 11A, and 11B is ticked, for example.In a case where the checkbox 1107 is ticked, the control unit 501determines to perform the left-right swapping, and the processingproceeds to step S810. In a case where the checkbox 1107 is not ticked,the processing proceeds to step S812.

In step S810, the control unit 501 generates a processed live view imageby swapping the positions of a right image and a left image in the liveview image acquired in step S808 based on the lens information acquiredin step S808 (left-right swapping). The control unit 501 generates theprocessed image by swapping the positions of the right image and theleft image in the live view image based on central coordinates includedin the lens information received together with the live view image (therespective optical axis centers of the left eye optical system 301L andthe right eye optical system 301R).

The processing of the left-right swapping will be described in detail.The control unit 501 acquires the central coordinates (the respectiveoptical axis centers of the left eye optical system 301L and the righteye optical system 301R) from the lens information acquired from thecamera 100 together with the live view image. The control unit 501generates a processed image by swapping the positions of a right imageand a left image in the captured image based on the central coordinates(left-right swapping). For example, the control unit 501 identifies aregion of the right image in the captured image based on the centralcoordinate of the right image, and identifies a region of the left imagein the captured image based on the central coordinate of the left image.Then, the control unit 501 swaps the positions of the identified tworegions. In the present exemplary embodiment, the right image and theleft image are arranged side by side in the left-right direction in thecaptured image. By the left-right swapping, a left-right positionalrelationship between the right image and the left image is inverted. Toidentify the regions of the right image and the left image moreaccurately, respective radii (diameters or radii) of the right image andthe left image may be acquired from information regarding the duallenses.

FIGS. 9A and 9B are schematic diagrams of the left-right swapping. FIG.9A illustrates conventional left-right swapping that does not useinformation regarding the dual lenses. FIG. 9B illustrates theleft-right swapping according to the present exemplary embodiment thatuses information regarding the dual lenses.

As illustrated in FIGS. 9A and 9B, in an image 901 not subjected to theleft-right swapping, a right image 903 being a circular fisheye image isarranged on the left side, and a left image 907 being a circular fisheyeimage is arranged on the right side.

In FIG. 9A, the image 901 is divided at a central coordinate 902 of theimage 901 into a left-half image and a right-half image, and thepositions of the left-half image and the right-half image are swapped.In other words, the left-half image is moved to the right side of theright-half image. An image 911 is an image after such left-rightswapping.

In FIG. 9A, the shift amount 906 is smaller than the shift amount 905.In other words, in the image 901, the right image 903 shifts to thecenter of the image 901 from the center of the left-half image of theimage 901. Similarly, the shift amount 910 is smaller than the shiftamount 909. In other words, in the image 901, the left image 907 shiftsto the center of the image 901 from the center of the right-half imageof the image 901. Thus, in the image 911, a central coordinate 913 ofthe left image 907 in the left-right direction is shifted from a centralcoordinate 904 by a distance 914, and a central coordinate 916 of theright image 903 in the left-right direction is shifted from a centralcoordinate 908 by a distance 917. Good stereoscopic view accordinglybecomes difficult.

In the present exemplary embodiment, by using lens information, in animage 837 (FIG. 9B) after the left-right swapping, a central coordinateof a left image in the left-right direction can be matched with thecentral coordinate 904, and a central coordinate of a right image in theleft-right direction can be matched with the central coordinate 908.Good stereoscopic view of the image 837 can be consequently obtained.

The method of the left-right swapping is not limited to theabove-described method. For example, the shift amounts 905, 906, 909,and 910 in FIG. 9A may be acquired from information regarding the duallenses, and when the positions of a right image and a left image areswapped, the right image and the left image may be arranged so that theacquired shift amounts are maintained, and remaining regions may befilled with black. The shift amount 905 is a distance from the left endof the captured image to the left end of the right image, and the shiftamount 906 is a distance from the center of the captured image to theright end of the right image. In the left-right swapping, the shiftamount 905 becomes a distance from the left end of the captured image tothe left end of the left image, and the shift amount 906 becomes adistance from the center of the captured image to the right end of theleft image. Similarly, the shift amount 909 is a distance from the rightend of the captured image to the right end of the left image, and theshift amount 910 is a distance from the center of the captured image tothe left end of the left image. In the left-right swapping, the shiftamount 909 becomes a distance from the right end of the captured imageto the right end of the right image, and the shift amount 910 becomes adistance from the center of the captured image to the left end of theright image.

In step S811, the control unit 501 displays the processed live viewimage generated in step S810 on the display unit 506.

In step S812, the control unit 501 displays the live view image acquiredin step S808 on the display unit 506. In other words, a live view imageoutput from the camera 100 is displayed as-is on the display unit 506.

In step S813, the control unit 501 determines whether to end the PC liveview. For example, in a case where connection between the camera 100 andthe PC 500 is canceled, or the user issues an end instruction of the PClive view to the camera 100 or the PC 500, the control unit 501determines to end the PC live view. The end instruction of the PC liveview is issued by a press of an end button 1108 illustrated in FIGS. 11Ato 11C, for example. In a case where it is determined that the PC liveview is to be ended (YES in step S813), the operation illustrated inFIG. 8 is ended. In a case where it is determined that the PC live viewis not to be ended (NO in step S813), the processing returns to stepS806.

As described above, in a case where the equirectangular display is to beperformed (NO in step S806), the processing proceeds to step S814 fromstep S806. In step S814, the control unit 501 transmits a live viewimage request to the camera connected in step S801. In the presentexemplary embodiment, the live view image request transmitted in stepS814 is a request for a live view image with low resolution (resolutionlower than the normal resolution). In a case where the equirectangulardisplay is performed, equirectangular conversion (conversion from acircular fisheye image into an equirectangular image) is to beperformed. As the resolution of an image to be subjected to theequirectangular conversion becomes higher, a time required for theequirectangular conversion increases, and a delay caused by theequirectangular conversion increases. In the present exemplaryembodiment, to speed up the equirectangular conversion (shortening thetime required for the equirectangular conversion), a request for a liveview image with low resolution is transmitted. If the delay caused bythe equirectangular conversion falls within an allowable range, arequest for a live view image with the normal resolution may betransmitted also in a case where the equirectangular display is to beperformed.

In step S815, the control unit 501 receives, from the camera connectedin step S801, a live view image captured by the camera and lensinformation regarding the dual lenses attached to the camera. Theresolution of the live view image received in step S815 is the lowresolution. The lens information received in step S815 is informationconverted to be suitable for the received live view image (for example,the lens information converted in step S608 of FIG. 6 ).

In step S816, the control unit 501 determines whether to execute theleft-right swapping. In a case where it is determined that theleft-right swapping is to be executed (YES in step S816), the processingproceeds to step S817. In a case where it is determined that theleft-right swapping is not to be executed (NO in step S816), theprocessing proceeds to step S819. In step S816, the control unit 501determines whether to execute the left-right swapping based on whetherthe checkbox 1107 illustrated in FIG. 11C is ticked, for example. In acase where the checkbox 1107 is ticked, the control unit 501 determinesto perform the left-right swapping, and the processing proceeds to stepS817. In a case where the checkbox 1107 is not ticked, the processingproceeds to step S819.

In step S817, based on the lens information acquired in step S815, thecontrol unit 501 swaps the positions of a right image and a left imagein the live view image acquired in step S815, and converts the rightimage and the left image into equirectangular images. The conversioninto an equirectangular image (equirectangular conversion) is conversionprocessing of converting an image in such a manner that a latitude line(horizontal line) and a longitude line (vertical line) orthogonallyintersect with each other while regarding a circular fisheye image as asphere, as in equidistant cylindrical projection of a map. By theequirectangular conversion, a circular fisheye image having a circularshape is converted into an equirectangular image having a rectangularshape.

The control unit 501 generates a map including pixels of a circularfisheye image and a conversion parameter that are to be used for drawingpixels in an equirectangular image. The map indicates a position in anunconverted image to which each pixel in a converted image corresponds.In the present exemplary embodiment, a map for equirectangularconversion is generated in such a manner that the positions of a rightimage and a left image can be corrected in addition to enabling acircular fisheye image to be converted into an equirectangular image. Inthe present exemplary embodiment, a map is generated so that theequirectangular conversion and the left-right swapping can besimultaneously performed. In addition, the control unit 501 may generatea map based on a lens design value corrected using an individual valueincluded in the lens information received together with the live viewimage.

The control unit 501 generates a processed image by performing theequirectangular conversion using the generated map. The left-rightswapping is performed as part of the equirectangular conversion, but theleft-right swapping may be performed separately from the equirectangularconversion.

FIG. 10 is a schematic diagram illustrating the equirectangularconversion including left-right swapping conversion according to thepresent exemplary embodiment. As illustrated in FIG. 10 , in an image1001 not subjected to the equirectangular conversion, a right image 1002being a circular fisheye image is arranged on the left side, and a leftimage 1005 being a circular fisheye image is arranged on the right side.

An image 1008 is an image after the equirectangular conversion, andincludes equirectangular images 1009 and 1010. In the present exemplaryembodiment, a map of the equirectangular conversion is generated so thatassociation as indicated by arrows 1011 and 1012 is performed. In themap of the present exemplary embodiment, pixels in the equirectangularimage 1009 arranged on the left side are associated with the respectivepositions in the left image 1005 arranged on the right side, and pixelsin the equirectangular image 1010 arranged on the right side areassociated with the respective positions in the right image 1002arranged on the left side. By using such a map, the left image 1005arranged on the right side is converted into the equirectangular image1009 arranged on the left side, and the right image 1002 arranged on theleft side is converted into the equirectangular image 1010 arranged onthe right side. In other words, in addition to the circular fisheyeimages being converted into the equirectangular images, the positions ofthe right image and the left image are swapped. Good stereoscopic viewis thereby enabled.

In step S818, the control unit 501 displays the processed live viewimage generated in step S817 on the display unit 506.

In step S819, the control unit 501 converts the right image and the leftimage in the live view image acquired in step S815 into equirectangularimages without swapping the positions of the right image and the leftimage. In other words, the control unit 501 generates a processed liveview image by performing the equirectangular conversion withoutperforming the left-right swapping.

In step S820, the control unit 501 displays the processed live viewimage generated in step S819 on the display unit 506.

In a case where a single lens is attached to the camera 100 (NO in stepS804), the processing in step S821 is performed. In step S821, thesystem control unit 50 transmits a live view image captured by a singlelens to the display unit 506. Because the processing in step S821 issimilar to conventional processing of displaying a live view imagecaptured by a single lens on a PC, a detailed description thereof willbe omitted.

In each of steps S810, S817, and S819, the control unit 501 executesimage processing on the live view image acquired from the connectedcamera. In step S813 subsequent to steps S810, S817, and S819, thecontrol unit 501 determines whether to end the PC live view. Then, in acase where the PC live view is to be continued (NO in step S813), theprocessing returns to step S806 antecedent to steps S810, S817, andS819. Thus, in the operation illustrated in FIG. 8 , the imageprocessing in any of steps S810, S817, and S819 may possibly berepeatedly executed.

Thus, to speed up the image processing, the control unit 501 may recordinformation regarding the executed image processing on the RAM 503, anduse the information in image processing to be executed next time orlater. For example, the control unit 501 records a correspondencerelationship between pixels not subjected to the image processing andpixels having been subjected to the image processing (image processingmap). The image processing map can continuously be used as long as theresolution of the live view image and the lens information stay thesame. When the control unit 501 executes the image processing in any ofsteps S810, S817, and S819, the control unit 501 records an imageprocessing map of the image processing. Then, when the control unit 501executes the same image processing again, the control unit 501 executesthe image processing using the recorded image processing map. With thisconfiguration, the speed of the image processing can be increased.

FIGS. 11A to 11C are schematic diagrams each illustrating an example ofdisplay (display of the PC live view) on an application screen to bedisplayed by the control unit 501 on the display unit 506. A screen 1100is an application screen (remote live view screen). The screen 1100includes a live view display region 1101, a guide display region 1102, aguide display region 1103, an operation region 1104, and the end button1108.

The live view display region 1101 is a region for displaying the liveview image. The live view display region 1101 includes a display region1101A on the left side and a display region 1101B on the right side. Theguide display region 1102 is a region for displaying a character stringindicating whether an image displayed in the display region 1101A on theleft side is an image of which of the two optical systems (the left eyeoptical system 301L and the right eye optical system 301R) of the duallenses. The guide display region 1103 is a region for displaying acharacter string indicating whether an image displayed in the displayregion 1101B on the right side is an image of which of the two opticalsystems (the left eye optical system 301L and the right eye opticalsystem 301R) of the dual lenses. The operation region 1104 is a regionfor receiving an operation related to the PC live view, and radiobuttons 1105 and 1106 and the checkbox 1107 are displayed in theoperation region 1104. The radio button 1105 is a radio button to beselected when the circular fisheye display is to be performed, and theradio button 1106 is a radio button to be selected when theequirectangular display is to be performed. In a case where the radiobutton 1105 is in the selected state, the radio button 1106 enters theunselected state. In a case where the radio button 1105 is in theunselected state, the radio button 1106 enters the selected state. Thecheckbox 1107 is a checkbox to be ticked when the left-right swapping isto be performed. If the checkbox 1107 is operated, the positions of aright image (right eye image) and a left image (left eye image) of alive view image are swapped, and the character strings displayed in theguide display regions 1102 and 1103 are swapped as well. The end button1108 is a button for ending the PC live view. An enlarged display button1109 is a button for issuing an execution instruction of enlargeddisplay processing. A frame 1110 is an item (enlargement frame)indicating a range to be enlarged (enlarged range) in the enlargeddisplay processing, on the live view image. By determining the enlargedrange by preliminarily operating the position of the frame 1110, andthen executing the enlarged display processing by pressing the enlargeddisplay button 1109, the user can display a desired range in an enlargedstate.

In FIG. 11A, the radio button 1105 for performing the circular fisheyedisplay is selected. The checkbox 1107 for performing the left-rightswapping is not ticked. Thus, the live view image acquired from a camerais displayed as-is in the live view display region 1101. Specifically,the right eye image being a circular fisheye image is displayed in thedisplay region 1101A on the left side, and the left eye image being acircular fisheye image is displayed in the display region 1101B on theright side.

In FIG. 11B, the radio button 1105 for performing the circular fisheyedisplay is selected, and the checkbox 1107 for performing the left-rightswapping is ticked. Thus, the positions of the right eye image and theleft eye image in the live view image acquired from a camera areswapped.

Then, a live view image after the left-right swapping is displayed inthe live view display region 1101. Specifically, the left eye imagebeing a circular fisheye image is displayed in the display region 1101Aon the left side, and the right eye image being a circular fisheye imageis displayed in the display region 1101B on the right side.

In FIG. 11C, the radio button 1106 for performing the equirectangulardisplay is selected, and the checkbox 1107 for performing the left-rightswapping is ticked. Thus, the positions of the right eye image and theleft eye image in the live view image acquired from a camera areswapped, and the right eye image and the left eye image (both beingcircular fisheye images) are converted into equirectangular images.Then, a live view image after the left-right swapping and theequirectangular conversion is displayed in the live view display region1101. Specifically, the left eye image being an equirectangular image isdisplayed in the display region 1101A on the left side, and the righteye image being an equirectangular image is displayed in the displayregion 1101B on the right side.

In a case where the PC live view image acquired from the camera 100 isan enlarged image, because a live view image obtained by extracting onlyan enlargement frame portion of a circular fisheye image is transmitted,image information for equirectangular conversion display cannot beobtained. For this reason, in a case where the equirectangularconversion display is to be performed, enlarged display is prohibited bybringing the enlarged display button 1109 into a disabled state.Alternatively, in a case where the enlarged display button 1109 ispressed, display is switched to the enlarged display by forciblyswitching the equirectangular conversion display to the circular fisheyedisplay.

<Description on Live View Enlarged Display Control>

In a case where the live view display is performed on the display unit506 by the PC live view, live view display of an enlarged image can beimplemented by transmitting an enlargement instruction of a dual-lensimage to the camera 100 from an application of the PC 500. At this time,enlargement processing of the image is executed by the camera 100. ThePC 500 receives a live view image to which the enlargement processinghas been applied, and displays the live view image on the display unit506. The PC 500 can issue execution and cancel instructions of theenlargement processing to the camera 100, and set a range (enlargedrange) of the dual-lens image to which the camera 100 applies theenlargement processing. The setting of the enlarged range is executed bymoving an enlargement frame 810 in response to a user operation on thelive view image displayed on the display unit 506, for example. Thecontrol unit 501 of the PC 500 notifies the camera 100 of the enlargedrange by transmitting information indicating the position of theenlargement frame 810 to the camera 100.

In the PC live view display, as described above, because the left-rightarrangement of the left eye image and the right eye image input from theleft and right optical systems of the lens unit 300 is inverted in adual-lens image, the control unit 501 can display the left eye image andthe right eye image after swapping their positions.

At this time, a coordinate on a screen at which the enlargement frame810 is displayed does not match a coordinate of the camera 100. Forexample, a case is described where the enlargement frame 810 is set on ascreen so that the center of the enlargement frame 810 matches thecenter of the left eye image displayed on the left side. In this case, acentral coordinate of the enlargement frame 810 set on the screencorresponds to the approximate center of the right eye image in acoordinate system of the imaging unit 211 of the camera 100. Thus, evenif information indicating the position of an enlargement frame in the PClive view is solely transmitted to the camera 100, the camera 100sometimes fails to be notified of the enlarged range intended by theuser.

FIGS. 12A, 12B, and 12C are diagrams each illustrating a display exampleof an application screen to be displayed by the control unit 501 on thedisplay unit 506 during the enlarged display of the PC live viewaccording to the present exemplary embodiment. Regions similar to thosein FIGS. 11A, 11B, and 11C are assigned the same reference numerals, anddescriptions thereof will be omitted.

An enlarged image to which the camera 100 has applied the enlargementprocessing of enlarging part of a dual-lens image is output to the PC500, and the enlarged image is displayed on the display unit 506 of thePC 500. At this time, if the enlarged image is solely displayed on thedisplay unit 506, the user cannot determine a region in the dual-lensimage to which the displayed image corresponds. Thus, in a case wherethe enlarged image is transmitted from the camera 100, the PC 500acquires information indicating the enlarged range in the dual-lensimage (target region of the enlargement processing) from the camera 100,and displays navigation display indicating the enlarged range in thedual-lens image on the display unit 506. The navigation display includesa navigation display region 1211 indicating a region corresponding tothe entire dual-lens image, and a frame 1214 indicating the enlargedrange in the dual-lens image. The navigation display region 1211includes a region 1212 indicating a region on the left side and a region1213 indicating a region on the right side in a case where an imagebefore enlargement is displayed in the PC live view display. By checkingthe navigation display, the user can recognize a region in the dual-lensimage to which the displayed enlarged image corresponds.

FIG. 12A is a diagram illustrating an enlarged display state displayedwhen the enlarged display button 1109 on an application is pressed in astate in which the live view image illustrated in FIG. 11A is displayed.

A guide display region 1202 is a region for displaying informationindicating an image corresponding to a displayed enlarged image,similarly to the guide display region 1102 for the left image and theguide display region 1103 for the right image. In the guide displayregion 1102, a character string such as “left eye image” or “right eyeimage” is displayed corresponding to an enlargement target.

A reduced display button 1209 is an operation member for ending enlargeddisplay and shifting to full display.

A left-right switch button 1210 is a button for inputting an instructionto switch an enlargement target from an image currently displayed in anenlarged state to the other image of a left eye image and a right eyeimage. Upon a press of the left-right switch button 1210, control isperformed in such a manner as to switch a target image of enlargementprocessing from the left image to the right image or from the rightimage to the left image and set, as an enlarged range, a rangecorresponding to the enlarged range before switching.

The navigation display region 1211 is a region for displaying a GUI(navigation display) indicating the enlarged range currently displayedin an enlarged state of the entire region of the dual-lens image.

FIG. 12B is an enlarged view of the navigation display. The navigationdisplay includes regions corresponding to a left eye image and a righteye image arranged in a dual-lens image before enlargement. The region1212 is a region corresponding to the right eye image in the dual-lensimage before the enlargement. The region 1213 is a region correspondingto the left eye image in the dual-lens image before the enlargement. Anitem indicating whether an image displayed in each region corresponds towhich of the left eye image and the right eye image is also displayed.In the present exemplary embodiment, the item is a character string suchas “left eye image” or “right eye image”.

The frame 1214 is display of an enlargement frame indicating a targetregion (enlarged range) of enlarged display in a dual-lens image. Basedon the position of the frame 1214 on the navigation display, the usercan recognize whether a region displayed in an enlarged state is aregion in a right eye image or a region in a left eye image. The usercan also recognize the position of the region displayed in an enlargedstate in each image.

FIG. 13 is a flowchart illustrating the display processing of anenlarged image executed in step S831.

In step S1301, the control unit 501 draws an enlarged image acquiredfrom the camera 100 in the live view display region 1101. At this time,the control unit 501 may display the enlarged image after applyingenlargement or reduction processing in such a manner that the size ofthe enlarged image suits the size in a vertical direction or the size ina horizontal direction of the live view display region 1101, or displaythe enlarged image in the same size centered in the live view displayregion 1101.

In step S1302, the control unit 501 executes the navigation displayindicating a region in a dual-lens image to which the enlarged imagecorresponds.

In step S1303, the control unit 501 determines whether the left-rightswapping is enabled.

The left-right swapping is processing of swapping the positions of aleft eye image and a right eye image in an image before enlargement.Whether to enable or disable the left-right swapping is controlled basedon the presence or absence of a tick in the checkbox 1107. Then, in acase where the control unit 501 determines that the left-right swappingis enabled (YES in step S1303), the processing proceeds to step 1304. Onthe other hand, in a case where the control unit 501 determines that theleft-right swapping is disabled (NO in step S1303), the processingproceeds to step S1305.

In step S1304, the control unit 501 executes the navigation displaycorresponding to a left-right swapped dual-lens image. FIG. 12Cillustrates the navigation display corresponding to the left-rightswapped dual-lens image. At this time, because left and right eyes arein the correct arrangement order, guide display is performed as-is.Nevertheless, because an enlargement position differs from a coordinateposition on an image sensor, the display of the enlargement position ischanged to a coordinate position obtained after swapping.

In step S1305, the control unit 501 executes the navigation displaycorresponding to a dual-lens image not subjected to the left-rightswapping as illustrated in FIG. 12B. As described above, in a dual-lensimage captured using dual lenses, a left eye image and a right eye imageare arranged in an inverted arrangement order. Such guide display isdisplayed in the navigation display region 1211.

At this time, a coordinate position on an image sensor is indicated asthe enlargement position.

In step S1306, the control unit 501 determines whether acurrently-displayed enlarged image is a right eye image. In a case wherethe enlarged image is the right eye image (YES in step S1306), theprocessing proceeds to step S1307. In step S1307, the control unit 501displays a character string indicating the right eye image in the guidedisplay region 1202. In step S1308, the control unit 501 acquires aposition from a center position (central coordinate) of the right eyeimage to a top-left coordinate of an enlargement frame.

In a case where it is determined in step S1306 that the enlarged imageis not the right eye image (NO in step S1306), the processing proceedsto step S1309. In step S1309, the control unit 501 displays a characterstring indicating the left eye image in the guide display region 1202.In step S1310, the control unit 501 acquires a position from a centerposition (central coordinate) of the left eye image to a top-leftcoordinate of the enlargement frame.

<Control of Enlargement Frame Movement Command Issued when Left andRight Eye Images are Swapped>

As described above, in a case where an enlarged image is displayed inthe PC live view display, by the navigation display, the user can benotified of a region of a dual-lens image to which the enlarged imagecorresponds. The PC 500 of the present exemplary embodiment can furtherdesignate, on the live view display, a target region (enlarged range) ofthe enlargement processing to be executed by the camera 100. The usercan determine the position of the enlarged range by moving anenlargement frame indicating the enlarged range that is displayed in thePC live view using an operation member (not illustrated). Even ifinformation indicating the position of the enlargement frame in the PClive view is solely transmitted to the camera 100, such an issue thatthe camera 100 fails to be notified of an enlarged range intended by theuser occurs.

As described above, in the dual-lens image transmitted by the camera100, a left eye image is arranged on the right side in the dual-lensimage and a right eye image is arranged on the left side in thedual-lens image. In the PC live view display, an image obtained byapplying the left-right swapping of swapping the left eye image and theright eye image of the dual-lens image acquired from the camera 100 issometimes displayed. In such a case, the user moves the enlargementframe indicating the enlarged range in the displayed image, but acoordinate system in the displayed image differs from a coordinatesystem in the camera 100. Specifically, it is assumed that the userdesignates an enlargement frame to be in a region on the left side inthe coordinate system of the displayed image to designate a portion inthe left eye image in an image in the PC live view display. On the otherhand, in the coordinate system of the dual-lens image processed by thecamera 100, the left eye image exists in a region on the right side inthe dual-lens image. Thus, if a coordinate indicating the position ofthe enlargement frame in the coordinate system of the PC live viewdisplay is transmitted to the camera 100 as-is, a position differentfrom the position intended by the user is designated in some cases.

In view of such an issue, the control unit 501 converts positioninformation of the enlarged range set on the PC live view display intoposition information in an image generated by the camera 100 based on adisplay setting (display mode, display format) of the PC live view, andoutputs the converted position information to the camera 100. Morespecifically, in a case where a target region of processing to beexecuted by the camera 100 is set on the live view display, the controlunit 501 converts the set position of the target region based on adisplay format of the live view display, and transmits an instruction tothe camera 100. With this configuration, in the PC live view display,even if image processing involving the movement of a portion of an imageis applied to a captured image (dual-lens image) acquired by the camera100, processing to be executed by the camera 100 can be applied to arange intended by the user.

As illustrated in FIG. 11B, in a case where a live view image isdisplayed on the PC 500 after swapping the positions of the left andright eye images, a coordinate system of the live view image displayedon the display unit 506 of the PC 500 and a coordinate system of a liveview image on the camera 100 are different. FIG. 14A is a schematicdiagram illustrating a coordinate system of the live view image on thecamera 100. As illustrated in FIG. 14A, in the coordinate system of thecamera 100, display control of live view is performed as a state inwhich the positions of a left eye image and a right eye image are notswapped.

The coordinate system illustrated in FIG. 14A corresponds to coordinatesystem information acquired from the camera 100, and the control unit501 acquires a coordinate plane of live view that corresponds to thesize of an image sensor of the camera 100, from the camera 100. A righteye lens coordinate region 1422 in dual lenses that is positioned on theleft side on the acquired live view coordinate plane, and a left eyelens coordinate region 1423 in dual lenses that is positioned on theright side on the live view coordinate plane exist.

The control unit 501 receives enlargement frame information indicatingthe position and the size of the enlargement frame 810, and therespective center positions of left and right fisheye images in lensinformation (a center 1420 of the right eye lens coordinate region 1422and a center 1421 of the left eye lens coordinate region 1423), from thecamera 100 together with a PC live view image.

In a case where a live view image is displayed in the live view displayon the PC 500 with the left-right swapping being enabled, the live viewimage is displayed on the display unit 506 in a state in which thepositions of the left eye image and the right eye image are swappedbased on the respective central coordinates of the left eye image andthe right eye image as illustrated in FIG. 11B.

FIG. 14B illustrates the arrangement of images displayed on the displayunit 506 in a case where the control unit 501 performs the live viewdisplay after swapping the positions of the left and right images. Asillustrated in FIG. 14B, on a live view coordinate plane of the camera100, a left eye image is displayed in the left side image display region1101A centered on the center 1420 of the right eye lens coordinateregion 1422. In addition, a right eye image is displayed in the rightside image display region 1101B centered on the center 1421 of the lefteye lens coordinate region 1423. In other words, a live view image isdisplayed in a state in which a left-right relationship between theimages is inverted in an X-axis direction with respect to the coordinatesystem of the camera. Furthermore, the enlargement frame 810 issimilarly displayed in a state in which the respective center positionsof the left and right fisheye images (the center 1420 of the right eyelens coordinate region 1422 and the center 1421 of the left eye lenscoordinate region 1423) are swapped.

In a state in which the left-right swapping is enabled in this manner, amovement command of the enlargement frame 810 is issued by the user tothe PC 500 as indicated by an arrow illustrated in FIG. 14B. If aninstruction to move the enlargement frame 810 to a coordinatecorresponding to display of an enlargement frame at a movementdestination ordered by the user as illustrated in FIG. 14B is directlytransmitted to the camera 100 as in a case where the left-right swappingis disabled, such an issue that the enlargement frame 810 is moved to aposition unintended by the user occurs.

In a case where the left-right swapping is enabled and the movementcommand of the enlargement frame 810 has been received from the user,the control unit 501 issues the movement command of the enlargementframe 810 to the camera 100 after converting a movement destinationcoordinate position of the enlargement frame 810 into a coordinateposition in a live view coordinate system of the camera 100 that isillustrated in FIG. 14C.

In the enlarged region movement control in the live view display that isto be executed by the camera 100 based on a command from the PC 500, thecamera 100 may be notified of a top-left coordinate of an enlargedregion, or a central coordinate of the enlarged region.

<Movement Processing Procedure Example of Enlargement Frame in PC LiveView on PC>

FIG. 15 is an explanatory diagram illustrating a flowchart inenlargement frame movement processing in a normal state in which a liveview image of the PC 500 is not enlarged.

The processing in the flowchart is started in a state in which anon-enlarged dual-lens image is received and a live view image isdisplayed on the display unit 506 in live view on the PC 500.

In step S1501, the control unit 501 determines whether an enlargementframe movement command has been issued from the user. In a case wherethe enlargement frame movement command has been issued (YES in stepS1501), the processing proceeds to step S1502. In a case where theenlargement frame movement command has not been issued (NO in stepS1501), the procedure of enlargement frame movement control is ended.

In step S1502, the control unit 501 determines whether a movementdestination display position of the enlargement frame 810 that isordered by the user exists in the display region 1101B of the right eyeimage in left-right swapped display illustrated in FIG. 14B. In a casewhere the movement destination position of the enlargement frame 810exists in the display region 1101B of the right eye image (YES in stepS1502), the processing proceeds to step S1504. In a case where themovement destination position of the enlargement frame 810 does notexist in the display region 1101B of the right eye image (NO in stepS1502), the processing proceeds to step S1503.

In step S1503, the control unit 501 determines whether the movementdestination display position of the enlargement frame 810 that isordered by the user exists in the display region 1101A of the left eyeimage that is illustrated in FIG. 14B.

In a case where the movement destination position of the enlargementframe 810 exists in the display region 1101A of the left eye image (YESin step S1503), the processing proceeds to step S1507. In a case wherethe movement destination position of the enlargement frame 810 exists inneither of regions of left and right circular fisheye images (NO in stepS1503), the processing proceeds to step S1510.

In step S1504, the control unit 501 determines whether the current liveview image display is the left-right swapped display. In a case wherethe live view image display is the left-right swapped display (YES instep S1504), the processing proceeds to step S1505. In a case where theleft-right swapping is not performed (NO in step S1504), the processingproceeds to step S1511. The control unit 501 determines whether anexecution instruction of the left-right swapping has been issued basedon a selection state of the checkbox 1107 illustrated in FIGS. 11A, 11B,and 11C.

In step S1505, the control unit 501 calculates an offset coordinate fromthe central coordinate of the right eye lens coordinate region 1422 thatis indicated by the center 1420 illustrated in FIG. 14B to theenlargement frame 810 moved by the user, records the offset coordinateon the RAM 503, and advances the processing to step S1506.

In step S1506, the control unit 501 calculates a movement destinationcoordinate by adding the offset coordinate of the enlargement frame 810moved by the user that is recorded on the RAM 503, to the centralcoordinate of the left eye lens coordinate region 1423 that is indicatedby the center 1421 illustrated in FIG. 14B, and advances the processingto step S1511.

In step S1507, the control unit 501 determines whether the current liveview image display is the left-right swapped display. In a case wherethe live view image display is the left-right swapped display (YES instep S1507), the processing proceeds to step S1508. In a case where theleft-right swapping is not performed (NO in step S1507), the processingproceeds to step S1511.

In step S1508, the control unit 501 calculates an offset coordinate fromthe central coordinate of the left eye lens coordinate region 1423 thatis indicated by the center 1421 illustrated in FIG. 14B to theenlargement frame 810 moved by the user, records the offset coordinateon the RAM 503, and advances the processing to step S1509.

In step S1509, the control unit 501 calculates a movement destinationcoordinate by adding the offset coordinate of the enlargement frame 810moved by the user that is recorded on the RAM 503, to the centralcoordinate of the right eye lens coordinate region 1422 that isindicated by the center 1420 illustrated in FIG. 14B, and advances theprocessing to step S1511.

In step S1510, the control unit 501 discards the movement command andends the movement operation.

In step S1511, the control unit 501 commands the camera 100 to move theenlargement frame 810 to the calculated movement destination coordinateof the enlargement frame 810.

Through the above-described procedure, the movement operation of theenlargement frame 810 ends.

<Movement Processing Procedure Example of Enlargement Position inEnlarged PC Live View>

FIG. 16 is a flowchart illustrating control to be executed when amovement command of an enlargement position is received from the user ina case where an enlarged image is displayed as a live view image asillustrated in FIGS. 12A, 12B, and 12C.

The processing in the flowchart is started in a state in which anenlarged live view image received by the PC 500 is displayed on thedisplay unit 506.

In step S1601, the control unit 501 determines whether a movementcommand of an enlarged region has been issued from the user. In a casewhere the movement command has been issued (YES in step S1601), theprocessing proceeds to step S1602. In a case where the movement commandhas not been issued (NO in step S1601), the processing proceeds to stepS1605.

In step S1602, the control unit 501 determines whether an amount ofmovement of an enlargement frame exceeds a current eye image in a casewhere the enlargement frame is moved in response to the movement commandissued by the user. In a case where the amount of movement of theenlargement frame exceeds the current eye image (YES in step S1602), theprocessing proceeds to step S1604. In a case where the amount ofmovement of the enlargement frame does not exceed the current eye image(NO in step S1602), the processing proceeds to step S1603.

In step S1603, the control unit 501 determines whether an enlargedregion of the live view thoroughly protrudes to the outside of acurrently-displayed circular fisheye region of dual lenses, and theenlarged region is displayed as a black region. In a case where theenlarged region thoroughly protrudes (YES in step S1603), the processingproceeds to step S1604. In a case where the enlarged region does notprotrude thoroughly (NO in step S1603), the processing proceeds to stepS1607.

In step S1604, the control unit 501 discards the movement command of anenlargement position in the live view display and ends the procedure.

In step S1605, the control unit 501 determines whether the instructionto move the enlarged region to the opposite eye image from the currenteye image is input. In a case where the instruction has been input (YESin step S1605), the processing proceeds to step S1606. In a case wherethe instruction has not been input (NO in step S1605), the procedure isended.

In step S1606, the control unit 501 calculates a movement destinationcoordinate by adding an offset amount of the enlargement frame to acentral coordinate of the enlarged region in an image to be enlarged,which is a right eye image or a left eye image, and advances theprocessing to step S1607.

In step S1607, the control unit 501 commands the camera 100 to move thelive view enlarged display position to the calculated coordinate of theenlargement position.

Through the above-described procedure, the movement operation of theenlargement position in the enlarged display ends.

The above-described various types of control described as beingperformed by the system control unit 50 may be performed by one piece ofhardware, or the entire apparatus may be controlled by a plurality ofpieces of hardware (e.g., a plurality of processors or circuits) sharingthe processing. Similarly, the above-described various types of controldescribed as being performed by the control unit 501 may be performed byone piece of hardware, or the entire apparatus may be controlled by aplurality of pieces of hardware (e.g., a plurality of processors orcircuits) sharing the processing.

The exemplary embodiments of the disclosure have been described indetail, but the disclosure is not limited to these specific exemplaryembodiments, and various configurations without departing from thespirit of the disclosure are also included in the exemplary embodimentsof the disclosure. Furthermore, each of the above-described exemplaryembodiments merely indicates an exemplary embodiment of the disclosure,and the exemplary embodiments can be appropriately combined.

An application example of the disclosure is not limited to cameras andPCs, and the disclosure can be applied to any electronic device that canhandle two images with parallax. For example the disclosure can beapplied to a personal digital assistance (PDA), a mobile phone terminal,a portable image viewer, a printing apparatus, a digital photo frame, amusic player, a game machine, and an electronic book reader. Thedisclosure can also be applied to a video player, a display device(including a projection device), a tablet terminal, a smartphone, anartificial intelligence (AI) speaker, a home appliance, and anin-vehicle device.

OTHER EMBODIMENTS

Embodiment(s) of the disclosure can also be realized by a computer of asystem or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiment(s) and/or that includes one ormore circuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiment(s), and by a method performed by the computer of the systemor apparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiment(s) and/or controllingthe one or more circuits to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, amemory card, and the like.

While the disclosure has been described with reference to exemplaryembodiments, it is to be understood that the disclosure is not limitedto the disclosed exemplary embodiments. The scope of the followingclaims is to be accorded the broadest interpretation so as to encompassall such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No.2021-091346, filed May 31, 2021, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information processing apparatus comprising: acommunication unit configured to communicate with an imaging apparatusconfigured to capture one third image including a first imagecorresponding to a first image input via a first optical system, and asecond image corresponding to a second image input via a second opticalsystem having predetermined parallax with respect to the first opticalsystem; a control unit configured to display the third image on adisplay unit; and a setting unit configured to set a position of atarget region to which predetermined image processing is to be appliedby the imaging apparatus, in the displayed third image, wherein thesetting unit converts the position of the target region set in the thirdimage displayed on the display unit based on a display format in whichthe control unit displays the third image on the display unit, andwherein the communication unit outputs the converted position of thetarget region to the imaging apparatus.
 2. The information processingapparatus according to claim 1, wherein the control unit displays thethird image in any of a plurality of display formats including a firstformat of displaying the third image including the first image and thesecond image with positions of the first image and the second imagebeing swapped, and a second format of displaying the third imageincluding the first image and the second image without the positions ofthe first image and the second image being swapped, and wherein, in acase where the third image is displayed in the first format and theposition of the target region is set in a region corresponding to thefirst image of the third image displayed on the display unit, thesetting unit converts the position in such a manner that the targetregion is set in the region corresponding to the first image of thecaptured third image.
 3. The information processing apparatus accordingto claim 1, wherein the setting unit converts a coordinate of the targetregion designated in an image displayed on the display unit based on adisplay manner.
 4. The information processing apparatus according toclaim 1, wherein, in a case where the control unit displays an imageobtained by applying the predetermined image processing to the thirdimage, on the display unit, the control unit displays display forindicating a position of the target region in the third image, on thedisplay unit.
 5. The information processing apparatus according to claim1, wherein the predetermined image processing is enlargement processingof enlarging part of the third image.
 6. The information processingapparatus according to claim 5, wherein, in a case where the controlunit displays the third image to which the enlargement processing hasbeen applied, the control unit displays information indicating whetherthe displayed image is a portion corresponding to the first image or aportion corresponding to the second image.
 7. The information processingapparatus according to claim 1, wherein lens information indicating afirst coordinate corresponding to an optical axis center of the firstoptical system and a second coordinate corresponding to an optical axiscenter of the second optical system in the third image is acquired, andwherein the setting unit converts the position of the target regionbased on the position of the target region with respect to a coordinatecorresponds to the first coordinate or the second coordinate in thethird image displayed on the display unit.
 8. The information processingapparatus according to claim 1, further comprising an acquisition unitconfigured to acquire information regarding the first optical system andthe second optical system, wherein the control unit displays, on thedisplay unit, an image obtained by executing image processing ofcorrecting positions of the first image and the second image in thethird image, based on the information.
 9. The information processingapparatus according to claim 8, wherein the information includesinformation regarding an optical axis center of the first optical systemin the third image and information regarding an optical axis center ofthe second optical system in the third image, and wherein thepredetermined image processing is swapping positions of the first imageand the second image in the third image based on the optical axiscenters of the first optical system and the second optical system in thethird image.
 10. The information processing apparatus according to claim8, wherein the information regarding the first optical system and thesecond optical system includes a design parameter of a lens unitincluding the first optical system and the second optical system, and aparameter unique to the lens unit, and wherein the predetermined imageprocessing is executed based on the design parameter and the uniqueparameter.
 11. The information processing apparatus according to claim8, wherein the third image is an image in which the first image and thesecond image are arranged side by side in a left-right direction, andwherein the predetermined image processing is inverting a left-rightpositional relationship between the first image and the second image inthe third image.
 12. The information processing apparatus according toclaim 1 wherein the first optical system and the second optical systemeach include a fisheye lens, and wherein the first image and the secondimage are circular fisheye images.
 13. A control method of aninformation processing apparatus, the control method comprising:communicating with an imaging apparatus configured to capture one thirdimage including a first image corresponding to a first image input via afirst optical system, and a second image corresponding to a second imageinput via a second optical system having predetermined parallax withrespect to the first optical system; displaying the third image on adisplay unit; and setting a position of a target region to whichpredetermined image processing is to be applied by the imagingapparatus, in the displayed third image, wherein the position of thetarget region set in the third image displayed on the display unit isconverted based on a display manner in which the third image isdisplayed on the display unit, and wherein the converted position of thetarget region is output to the imaging apparatus.
 14. The control methodaccording to claim 13, further comprising displaying the third image inany of a plurality of display formats including a first format ofdisplaying the third image including the first image and the secondimage with positions of the first image and the second image beingswapped, and a second format of displaying the third image including thefirst image and the second image without the positions of the firstimage and the second image being swapped, wherein, in a case where thethird image is displayed in the first format and the position of thetarget region is set in a region corresponding to the first image of thethird image displayed on the display unit, the setting converts theposition in such a manner that the target region is set in the regioncorresponding to the first image of the captured third image.
 15. Thecontrol method according to claim 13, wherein the setting converts acoordinate of the target region designated in an image displayed on thedisplay unit based on a display manner.
 16. The control method accordingto claim 13, wherein, in a case where the displaying displays an imageobtained by applying the predetermined image processing to the thirdimage, on the display unit, the displaying displays display forindicating a position of the target region in the third image, on thedisplay unit.
 17. A non-transitory computer-readable storage mediumstoring a program for causing a computer to perform a control method,the method comprising: communicating with an imaging apparatusconfigured to capture one third image including a first imagecorresponding to a first image input via a first optical system, and asecond image corresponding to a second image input via a second opticalsystem having predetermined parallax with respect to the first opticalsystem; displaying the third image on a display unit; and setting aposition of a target region to which predetermined image processing isto be applied by the imaging apparatus, in the displayed third image,wherein the position of the target region set in the third imagedisplayed on the display unit is converted based on a display manner inwhich the third image is displayed on the display unit, and wherein theconverted position of the target region is output to the imagingapparatus.
 18. The non-transitory computer-readable storage mediumaccording to claim 17, further comprising displaying the third image inany of a plurality of display formats including a first format ofdisplaying the third image including the first image and the secondimage with positions of the first image and the second image beingswapped, and a second format of displaying the third image including thefirst image and the second image without the positions of the firstimage and the second image being swapped, wherein, in a case where thethird image is displayed in the first format and the position of thetarget region is set in a region corresponding to the first image of thethird image displayed on the display unit, the setting converts theposition in such a manner that the target region is set in the regioncorresponding to the first image of the captured third image.
 19. Thenon-transitory computer-readable storage medium according to claim 17,wherein the setting converts a coordinate of the target regiondesignated in an image displayed on the display unit based on a displaymanner.
 20. The non-transitory computer-readable storage mediumaccording to claim 17, wherein, in a case where the displaying displaysan image obtained by applying the predetermined image processing to thethird image, on the display unit, the displaying displays display forindicating a position of the target region in the third image, on thedisplay unit.